[graph_trainer] Add reusable dI/dW backward-node classification - #6
Draft
wolfcomos wants to merge 1 commit into
Draft
[graph_trainer] Add reusable dI/dW backward-node classification#6wolfcomos wants to merge 1 commit into
wolfcomos wants to merge 1 commit into
Conversation
Generalize the GraphPP dI/dW dependency analysis into a standalone helper the EP-overlap scheduler can consume: partition_backward_nodes classifies every node in the joint graph's gradient closures into di_nodes / dw_only_nodes / shared_nodes by pure ancestor algebra (closure of the input-gradient outputs vs closure of the parameter-gradient outputs), never inspecting op names or dtypes, and exposes movable_nodes, the subset of dw-only work a pass may DEFER (move later, never earlier). Safety rules keep deferral provable: - pinned per node: collectives, mutations, CPU synchronizations (_local_scalar_dense or device-to-host reads), side-effectful ops, and producers with unclassified LIVE users; a pure dead subtree (e.g. the unused getitem outputs of a multi-output quantize op) does not pin its producer. - mutation hazards: a reader of a mutated buffer's alias family that sits BEFORE the write without a data edge through it is pinned (deferring it could push the read past the write); readers downstream of the write or after it keep their order against the fixed write. A write whose target cannot be resolved to graph values empties movable_nodes entirely. split_di_dw.py stays untouched: its extraction-based dI membership deliberately includes dW-only placeholders as pass-through live-ins, and rebasing it on the closure helper would change which values it saves. Validated on a real tiny-MXFP8-GroupedExperts joint trace (EP mesh wired, world size 1): all wgrad GEMMs and their quantize/rearrange chains classify movable, dgrad chains stay on the dI side, and collectives are never movable. 13 CPU unit tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR 1 of 2 (stack: this →
graphpoc/deferred-dw)Generalize the GraphPP dI/dW dependency analysis into a standalone helper the EP-overlap scheduler can consume:
partition_backward_nodesclassifies every node in the joint graph's gradient closures intodi_nodes/dw_only_nodes/shared_nodesby pure ancestor algebra (closure of the input-gradient outputs vs closure of the parameter-gradient outputs), never inspecting op names or dtypes, and exposesmovable_nodes, the subset of dW-only work a pass may DEFER (move later, never earlier).Safety rules keep deferral provable:
_local_scalar_denseor device-to-host reads), side-effectful ops, and producers with unclassified LIVE users; a pure dead subtree (e.g. the unused getitem outputs of a multi-output quantize op) does not pin its producer.movable_nodesentirely.split_di_dw.pystays untouched: its extraction-based dI membership deliberately includes dW-only placeholders as pass-through live-ins, and rebasing it on the closure helper would change which values it saves.Validation
movable == dw_only(48).index_put_-into-new_emptycombine scatter is handled correctly).🤖 Generated with Claude Code